Evaluation of a Dutch stemming algorithm

In state of the art Information Retrieval (IR) systems the most salient problem is to improve recall rates while retaining high precision. A simple recall enhancing technique which can be useful for even the simplest boolean retrieval systems is stemming. It is obvious that an information-seeker who is looking for texts about, for example, dogs is probably interested in a text which contains the word dog. An algorithm which maps different morphological variants to their base form (stem) is called a stemming algorithm. The underlying assumption for fruitful usage of such a stemmer is that morphological variants of words are semantically related. This is obviously not always true. In information retrieval, the use of stemming is controversial (cf. Harman 1991). However, several authors (Frakes and Baeza-Yates 1992; Krovetz 1993; Popovic̆ and Willett 1992) report favourable results1. The UPLIFT project2 investigates whether linguistic tools can improve the performance of an IR system for Dutch texts. As a first step we will adapt and test two common stemming techniques for Dutch text. The first technique, quite popular in several experimental and commercial IR systems is suffix stripping. Suffix stripping is a pragmatic approach. The algorithms are small and efficient and are not hampered by linguistic claims. Efficiency is an important property of every subpart of an IR system, especially for modern interactive systems. However, the simple architecture of such algorithms has drawbacks, such as the easy introduction of errors. The second technique, Stemming based on morphological analysis, requires more complex resources. This approach tries to exploit linguistic knowledge about the internal structure of word forms. A necessary component for such a morphological analysis is a dictionary. In general, each word which has to be stemmed will involve dictionary lookup and therefore this technique will be considerably slower than suffix stripping. On the other hand, such careful